Skip to content

[FSDP] FSDP Support Muon and CSA - #79781

Merged
Xing-lil merged 7 commits into
PaddlePaddle:developfrom
Xing-lil:fsdp_supp_muon_csa
Oct 12, 2026
Merged

Xing-lil merged 7 commits into
PaddlePaddle:developfrom
Xing-lil:fsdp_supp_muon_csa

Conversation

@Xing-lil

@Xing-lil Xing-lil commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

New features

Description

FSDP Support Muon and CSA
Check in DSV4 with N4C32_DSV4_PT_seq32K_gbs32_exp256 (FSDP vs PP):

Strategy tokens/s/card max_mem_reserved
EP8+PP4 2303.24 230.79
EP8+FSDP32 2292.70 242.99

Check in DSV4 with N1C8 (FSDP vs Sharding Stage1):
1e790a26cc0b0e456598089e646ed587

是否引起精度变化

否

out.append((param, grad))
return out

def _muon_sharded_params_grads(self, group):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1

本 PR 新增了 FSDP + Muon 的整套支持,其中按矩阵边界跨 rank 切分(_muon_3d_shard_numel、_muon_sharded_params_grads 以及 broadcast_muon_params 的 muon-shard 分支)配合 ns_per_matrix 的逐矩阵 Newton-Schulz,是一条会直接影响权重更新数值的新分布式路径;但本次改动未新增任何测试,现有 test/collective/fleet/dygraph_group_fsdp.py 也未覆盖 Muon。

作者已在 muon.py 的 ns_per_matrix 文档中说明按 batch 切分会改变结果(bf16 下约 1.2e-2 相对误差),因此该切分路径的正确性与数值一致性需要多卡测试来保证。建议补充代表性的多卡用例:3D Muon 参数按矩阵边界跨 rank 切分,校验切分后最终权重与单卡/复制路径一致;并覆盖对齐回退(shard_align_numel 不满足时退回复制)与 _maybe_force_per_matrix_ns 自动开启 ns_per_matrix 的分支。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在最新提交中补充多卡测试 run_muon_shard(test/collective/fleet/dygraph_group_fsdp.py,经 test_dygraph_group_fsdp 以 2 卡启动):对 3D Muon 参数分别走矩阵边界切分路径与 owner 聚合的 dense 切分路径,单步训练后按 bitwise 校验两者最终权重完全一致,并断言 muon_shard_numel == HIDDEN*INTER、is_sharded、muon_owner_rank is None 以及自动开启 ns_per_matrix。该路径的数值一致性已有回归覆盖,此项已解决。

@Paddle-Review-Bot

Paddle-Review-Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态
1 FSDP+Muon 矩阵边界切分路径缺少多卡测试 P1 仓库规则:测试质量与验证 ✅

Powered by Nyanpasu claude with Opus 4.8 默认推理级别, please check the suggestions carefully.

@codecov-commenter

codecov-commenter commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.86992% with 30 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@4793e33). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...thon/paddle/distributed/fsdp/fully_shard_fusion.py 92.02% 26 Missing ⚠️
python/paddle/optimizer/muon.py 89.74% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop   #79781   +/-   ##
==========================================
  Coverage           ?   91.86%           
==========================================
  Files              ?        4           
  Lines              ?      369           
  Branches           ?        0           
==========================================
  Hits               ?      339           
  Misses             ?       30           
  Partials           ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

if hasattr(fsdp_context, "bind_decay_param_fun"):
fsdp_context.bind_decay_param_fun(self)
if hasattr(fsdp_context, "bind_optimizer"):
fsdp_context.bind_optimizer(self)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Muon 重写了 _apply_optimize,这部分逻辑在 Muon 中可能不会被调用?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

感谢,这块是给优化器 save/load 用的,之后会在 muon.py 补上对应调用。

full = getattr(param, "_muon_full_shape", None)
if full is None:
continue
param.get_tensor()._set_dims(full)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L1455 是想要对 local rank Tensor 设置了 global logical shape 吗?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是的,更新期间参数是本地分片的状态,更新完恢复成完整的 global logical shape。

@liym27 liym27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Xing-lil
Xing-lil merged commit d3b5b0c into PaddlePaddle:develop Oct 12, 2026
102 of 103 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants